Image display system, image display method, and image display program

ABSTRACT

Moving images are switched naturally when development of a story is switched according to a situation. An image display system includes a display unit that displays a moving image, an acquisition unit that acquires a motion of a user viewing the moving image displayed by the display unit, a determination unit that determines whether or not the motion of the user acquired by the acquisition unit satisfies a predetermined switching condition in which the user does not recognize switching between images even when the moving image displayed by the display unit is switched from first moving image data to second moving image data different from the first moving image, and a switching unit that switches the moving image displayed by the display unit from the first moving image data to the second moving image data when the determination unit determines that the switching condition is satisfied.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image display system, an imagedisplay method, and an image display program, and more particularly, toan image display system, an image display method, and an image displayprogram using a head mounted display.

Description of Related Art

In recent years, with the development of moving image capturingtechnology and recording media, flexibility in reproduction methods isalso being required. For example, a technology for preventinginterruption or disturbance of a reproduced video from occurring whenbranching from a branch portion of a stem story to a plurality of branchscenes is performed in a case in which data of a plurality of stories orscenes is recorded has been developed (see Japanese Unexamined PatentApplication Publication Nos. H10-27461 and H10-255443).

Meanwhile, a video of a plurality of scenes is prepared and developmentof the story is switched according to a viewing situation of a user,such that the user can enjoy a story suitable for the user. Here, whenthe development of the story is switched, the user can recognize thatthe moving image has been switched in a case in which switching of amoving image is unnatural. When the switching of the moving image isrecognized, the user, for example, does not concentrate on the story andcannot fully enjoy the story.

SUMMARY OF THE INVENTION

The present invention has been made in view of the above problems, andan object of the present invention is to prepare a video of a pluralityof scenes and switch between moving images naturally when development ofa story is switched according to a situation.

In order to resolve the above problem, an image display system accordingto the present invention includes a display unit that displays a movingimage; an acquisition unit that acquires a motion of a user viewing themoving image displayed by the display unit; a determination unit thatdetermines whether or not the motion of the user acquired by theacquisition unit satisfies a predetermined switching condition in whichthe user does not recognize switching between images even when themoving image displayed by the display unit is switched from first movingimage data to second moving image data different from the first movingimage; and a switching unit that switches the moving image displayed bythe display unit from the first moving image data to the second movingimage data when the determination unit determines that the switchingcondition is satisfied.

According to the present invention, it is possible to switch betweenmoving images naturally when development of a story is switchedaccording to a situation.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 is an external view illustrating a state in which a user wears ahead mounted display;

FIG. 2 is a perspective view schematically illustrating an overview ofan image display system of the head mounted display;

FIG. 3 is a diagram schematically illustrating an optical configurationof an image display system of the head mounted display;

FIG. 4 is a block diagram illustrating a configuration of an imagedisplay system;

FIG. 5 is a schematic diagram illustrating calibration for detection ofa gaze direction;

FIG. 6 is a schematic diagram illustrating position coordinates of acornea of a user;

FIG. 7 is a data configuration diagram illustrating a configuration ofdata for use in the image display system;

FIG. 8 is a flowchart illustrating an operation of the image displaysystem;

FIG. 9 illustrates an example of an image illustrating an imagedisplayed in the image display system.

FIG. 10 is a time chart illustrating a switching timing for a movingimage in the image display system.

FIG. 11 is a circuit diagram illustrating a circuit configuration of theimage display system.

DETAILED DESCRIPTION OF THE INVENTION <Configuration>

FIG. 1 is a diagram schematically illustrating an overview of a imagedisplay system 1. The image display system 1 includes a head mounteddisplay 100 and a gaze detection device 200. As illustrated in FIG. 1,the head mounted display 100 is mounted on a head of a user 300 for use.

The gaze detection device 200 detects a gaze direction of at least oneof right and left eyes of the user wearing the head mounted display 100,and specifies a focal point of the user, that is, a gaze point of theuser in a three-dimensional image displayed on the head mounted display.Further, the gaze detection device 200 also functions as a videogeneration device that generates videos displayed by the head mounteddisplay 100. For example, the gaze detection device 200 is a devicecapable of reproducing videos of stationary game machines, portable gamemachines, PCs, tablets, smartphones, phablets, video players, TVs, orthe like, but the present invention is not limited thereto. The gazedetection device 200 is wirelessly or wiredly connected to the headmounted display 100. In the example illustrated in FIG. 1, the gazedetection device 200 is wirelessly connected to the head mounted display100. The wireless connection between the gaze detection device 200 andthe head mounted display 100 can be realized using a known wirelesscommunication technique such as Wi-Fi (registered trademark) orBluetooth (registered trademark). For example, transfer of videosbetween the head mounted display 100 and the gaze detection device 200is executed according to a standard such as Miracast (registeredtrademark), WiGig (registered trademark), or WHDI (registeredtrademark).

FIG. 1 illustrates an example in which the head mounted display 100 andthe gaze detection device 200 are different devices. However, the gazedetection device 200 may be built into the head mounted display 100.

The head mounted display 100 includes a housing 150, a fitting harness160, and headphones 170. The housing 150 houses an image display system,such as an image display element, for presenting videos to the user 300,and a wireless transfer module (not illustrated) such as a Wi-Fi moduleor a Bluetooth (registered trademark) module. The fitting harness 160 isused to mount the head mounted display 100 on the head of the user 300.The fitting harness 160 may be realized by, for example, a belt or anelastic band. When the user 300 wears the head mounted display 100 usingthe fitting harness 160, the housing 150 is arranged at a position wherethe eyes of the user 300 are covered. Thus, when the user 300 wears thehead mounted display 100, a field of view of the user 300 is covered bythe housing 150.

The headphones 170 output audio for the video that is reproduced by thegaze detection device 200. The headphones 170 may not be fixed to thehead mounted display 100. Even when the user 300 wears the head mounteddisplay 100 using the fitting harness 160, the user 300 may freelyattach or detach the headphones 170.

FIG. 2 is a perspective diagram illustrating an overview of the imagedisplay system 130 of the head mounted display 100. Specifically, FIG. 2illustrates a region of the housing 150 that faces corneas 302 of theuser 300 when the user 300 wears the head mounted display 100.

As illustrated in FIG. 2, a convex lens 114 a for the left eye isarranged at a position facing the cornea 302 a of the left eye of theuser 300 when the user 300 wears the head mounted display 100.Similarly, a convex lens 114 b for a right eye is arranged at a positionfacing the cornea 302 b of the right eye of the user 300 when the user300 wears the head mounted display 100. The convex lens 114 a for theleft eye and the convex lens 114 b for the right eye are gripped by alens holder 152 a for the left eye and a lens holder 152 b for the righteye, respectively.

Hereinafter, in this specification, the convex lens 114 a for the lefteye and the convex lens 114 b for the right eye are simply referred toas a “convex lens 114” unless the two lenses are particularlydistinguished. Similarly, the cornea 302 a of the left eye of the user300 and the cornea 302 b of the right eye of the user 300 are simplyreferred to as a “cornea 302” unless the corneas are particularlydistinguished. The lens holder 152 a for the left eye and the lensholder 152 b for the right eye are referred to as a “lens holder 152”unless the holders are particularly distinguished.

A plurality of infrared light sources 103 are included in the lensholders 152. For the purpose of brevity, in FIG. 2, the infrared lightsources that irradiate the cornea 302 a of the left eye of the user 300with infrared light are collectively referred to as infrared lightsources 103 a, and the infrared light sources that irradiate the cornea302 b of the right eye of the user 300 with infrared light arecollectively referred to as infrared light sources 103 b. Hereinafter,the infrared light sources 103 a and the infrared light sources 103 bare referred to as “infrared light sources 103” unless the infraredlight sources 103 a and the infrared light sources 103 b areparticularly distinguished. In the example illustrated in FIG. 2, sixinfrared light sources 103 a are included in the lens holder 152 a forthe left eye. Similarly, six infrared light sources 103 b are includedin the lens holder 152 b for the right eye. Thus, the infrared lightsources 103 are not directly arranged in the convex lenses 114, but arearranged in the lens holders 152 that grip the convex lenses 114, makingthe attachment of the infrared light sources 103 easier. This is becausemachining for attaching the infrared light sources 103 is easier thanfor the convex lenses 114 that are made of glass or the like since thelens holders 152 are typically made of a resin or the like.

As described above, the lens holders 152 are members that grip theconvex lenses 114. Therefore, the infrared light sources 103 included inthe lens holders 152 are arranged around the convex lenses 114. Althoughthere are six infrared light sources 103 that irradiate each eye withinfrared light herein, the number of the infrared light sources 103 isnot limited thereto. There may be at least one light source 103 for eacheye, and two or more light sources 103 are desirable.

FIG. 3 is a schematic diagram of an optical configuration of the imagedisplay system 130 contained in the housing 150, and is a diagramillustrating a case in which the housing 150 illustrated in FIG. 2 isviewed from a side surface on the left eye side. The image displaysystem 130 includes infrared light sources 103, an image display element108, a hot mirror 112, the convex lenses 114, a camera 116, and a firstcommunication unit 118.

The infrared light sources 103 are light sources capable of emittinglight in a near-infrared wavelength region (700 nm to 2500 nm range).Near-infrared light is generally light in a wavelength region ofnon-visible light that cannot be observed by the naked eye of the user300.

The image display element 108 displays an image to be presented to theuser 300. The image to be displayed by the image display element 108 isprovided from the gaze detection device 200. The image display element108 can be realized by using an existing liquid crystal display (LCD) ororganic electro luminescence display (OLED).

The hot mirror 112 is arranged between the image display element 108 andthe cornea 302 of the user 300 when the user 300 wears the head mounteddisplay 100. The hot mirror 112 has a property of transmitting visiblelight created by the image display element 108, but reflectingnear-infrared light.

The convex lenses 114 are arranged on the opposite side of the imagedisplay element 108 with respect to the hot mirror 112. In other words,the convex lenses 114 are arranged between the hot mirror 112 and thecornea 302 of the user 300 when the user 300 wears the head mounteddisplay 100. That is, the convex lenses 114 are arranged at positionsfacing the corneas 302 of the user 300 when the user 300 wears the headmounted display 100.

The convex lenses 114 condense image display light that is transmittedthrough the hot mirror 112. Thus, the convex lenses 114 function asimage magnifiers that enlarge an image created by the image displayelement 108 and present the image to the user 300. Although only one ofeach convex lens 114 is illustrated in FIG. 2 for convenience ofdescription, the convex lenses 114 may be lens groups configured bycombining various lenses or may be a plano-convex lens in which onesurface has curvature and the other surface is flat.

A plurality of infrared light sources 103 are arranged around the convexlens 114. The infrared light sources 103 emit infrared light toward thecornea 302 of the user 300.

Although not illustrated in the figure, the image display system 130 ofthe head mounted display 100 includes two image display elements 108 andcan independently generate an image to be presented to the right eye ofthe user 300 and an image to be presented to the left eye of the user.Accordingly, the head mounted display 100 can present a parallax imagefor the right eye and a parallax image for the left eye to the right andleft eyes of the user 300. Thereby, the head mounted display 100 canpresent a stereoscopic video that has a feeling of depth to the user300.

The infrared light reaching the cornea 302 of the user 300 is reflectedby the cornea 302 of the user 300 and is directed to the convex lens 114again. This infrared light is transmitted through the convex lens 114and is reflected by the hot mirror 112. The camera 116 includes a filterthat blocks visible light and images the near-infrared light reflectedby the hot mirror 112. That is, the camera 116 is a near-infrared camerawhich images the near-infrared light emitted from the infrared lightsources 103 and reflected by the cornea of the eye of the user 300.

Although not illustrated in the figure, the image display system 130 ofthe head mounted display 100 includes two cameras 116, that is, a firstimaging unit that captures an image including the infrared lightreflected by the right eye and a second imaging unit that captures animage including the infrared light reflected by the left eye. Thereby,images for detecting gaze directions of both the right eye and the lefteye of the user 300 can be acquired.

The first communication unit 118 outputs the image captured by thecamera 116 to the gaze detection device 200 that detects the gazedirection of the user 300. Specifically, the first communication unit118 transmits the image captured by the camera 116 to the gaze detectiondevice 200. Although the gaze detection unit 202 functioning as a gazedirection detection unit will be described below in detail, the gazedirection unit is realized by an image display program executed by acentral processing unit (CPU) of the gaze detection device 200. When thehead mounted display 100 includes computational resources such as a CPUor a memory, the CPU of the head mounted display 100 may execute theprogram that realizes the gaze direction detection unit.

As will be described below in detail, bright spots caused bynear-infrared light reflected by the cornea 302 of the user 300 and animage of the eyes including the cornea 302 of the user 300 observed in anear-infrared wavelength region are captured in the image captured bythe camera 116.

Although the configuration for presenting the image to the left eye ofthe user 300 in the image display system 130 has mainly been describedabove, a configuration for presenting an image to the right eye of theuser 300 is the same as above.

FIG. 4 is a block diagram of the head mounted display 100 and the gazedetection device 200 according to the image display system 1. Asillustrated in FIG. 4, and as described above, the image display system1 includes the head mounted display 100 and the gaze detection device200 which communicate with each other.

As illustrated in FIG. 4, the head mounted display 100 includes thefirst communication unit 118, a display unit 121, an infrared lightirradiation unit 122, an image processing unit 123, an imaging unit 124,a motion sensor 125, and a position sensor 126.

The first communication unit 118 is a communication interface having afunction of communicating with a second communication unit 22 of thegaze detection device 200. As described above, the first communicationunit 118 communicates with the second communication unit 22 throughwired or wireless communication. Examples of usable communicationstandards are as described above. The first communication unit 118transmits image data to be used for gaze detection transferred from theimaging unit 124 or the image processing unit 123 to the secondcommunication unit 22. Further, the first communication unit 118transfers image data or a marker image transmitted from the gazedetection device 200 to the display unit 121. The image data is, forexample, moving image data with a story. For example, the image data maybe a moving image such as a movie, may be a moving image such as a videogame, or may be a moving image such as a landscape video. Further, theimage data may be a parallax image pair including a right-eye parallaximage and a left-eye parallax image for displaying a three-dimensionalimage. Further, the image data may be moving image data of an all skyimage of 360°.

The display unit 121 has a function of displaying the image datatransferred from the first communication unit 118 on the image displayelement 108. Further, the display unit 121 displays a marker imageoutput from a display processing unit 201 at designated coordinates ofthe image display element 108.

The infrared light irradiation unit 122 controls the infrared lightsources 103 and irradiates the right eye or the left eye of the userwith infrared light.

The image processing unit 123 performs image processing on the imagecaptured by the imaging unit 124, as necessary, and transfers aprocessed image to the first communication unit 118.

The imaging unit 124 captures an image of near-infrared light reflectedby each eye using the camera 116. Further, the imaging unit 124 capturesan image including the eyes of the user gazing at the marker imagedisplayed on the image display element 108. The imaging unit 124transfers the image obtained by the imaging to the first communicationunit 118 or the image processing unit 123.

The motion sensor 125 is a sensor that detects a slope of the headmounted display 100, that is, a slope of the head of the user. Themotion sensor 125 is realized by, for example, an acceleration sensor ora gyro sensor. The motion sensor 125 sequentially detects the slope ofthe head mounted display 100 and transfers the slope to the firstcommunication unit 118. For example, when the motion sensor 125 is agyro sensor, the motion sensor 125 transfers a rotation angle (roll,pitch, and yaw) of the head mounted display 100 to the firstcommunication unit 118. It should be noted that the rotation angle istransmitted to the gaze detection device 200 via the first communicationunit 118.

The position sensor 126 is a sensor that detects a position of the headmounted display 100, that is, a position of the head of the user wearingthe head mounted display 100. The position sensor 126 sequentiallydetects three-dimensional coordinates (x, y, z) as a position of thehead mounted display 100 and transfers the three-dimensional coordinatesto the first communication unit 118. It should be noted that thethree-dimensional coordinates are transmitted to the gaze detectiondevice 200 via the first communication unit 118.

The above is the description of the configuration of the head mounteddisplay 100.

As illustrated in FIG. 4, the gaze detection device 200 includes acentral processing unit (CPU) 20, a storage device 21, the secondcommunication unit 22, an input device 23, and an output device 24.

The storage device 21 is a recording medium that stores various programsand data that is required for an operation of the gaze detection device200. Specifically, the storage device 21 stores image data 211 to beprovided to the head mounted display 100, determination condition data212 as to whether or not to execute a determination for image switching,switching condition data 213 that is an image switching condition, andan image display program P that realizes the gaze detection device 200.

In the storage device 21, image data 211 for a plurality of patterns ofstories associated with the switching condition is stored. In addition,by reading and executing the image display program P stored in thestorage device 21, the CPU 20 executes processes of the displayprocessing unit 201, the gaze detection unit 202, the determination unit203, the switching unit 204, and the difference calculation unit 205. Itshould be noted that the storage device 21 is realized by, for example,a hard disk drive (HDD) or a solid state drive (SSD).

Next, detection of a gaze direction will be described.

FIG. 5 is a schematic diagram illustrating calibration for detection ofthe gaze direction. The gaze direction of the user 300 is realized bythe gaze detection unit 202 in the gaze detection device 200 analyzingthe video captured by the camera 116 and output to the gaze detectiondevice 200 by the first communication unit 118.

The display processing unit 201 causes the image display element 108 ofthe head mounted display 100 to display nine points (marker images)including points Q₁ to Q₉ as illustrated in FIG. 5. The gaze detectiondevice 200 causes the user 300 to sequentially gaze at the points Q₁ upto Q₉. In this case, the user 300 is requested to gaze at each of thepoints by moving his or her eyeballs as much as possible without movinghis or her neck. The camera 116 captures images including the cornea 302of the user 300 when the user 300 is gazing at the nine points includingthe points Q₁ to Q₉.

FIG. 6 is a schematic diagram illustrating the position coordinates ofthe cornea 302 of the user 300. The gaze detection unit 202 in the gazedetection device 200 analyzes the images captured by the camera 116 anddetects bright spots 105 derived from the infrared light. When the user300 gazes at each point by moving only his or her eyeballs, thepositions of the bright spots 105 are considered to be stationaryregardless of the point at which the user gazes. Thus, on the basis ofthe detected bright spots 105, the gaze detection unit 202 sets atwo-dimensional coordinate system 306 in the image captured by thecamera 116.

Further, the gaze detection unit 202 detects the center P of the cornea302 of the user 300 by analyzing the image captured by the camera 116.This is realized by using known image processing such as the Houghtransform or an edge extraction process. Accordingly, the gaze detectionunit 202 can acquire the coordinates of the center P of the cornea 302of the user 300 in the set two-dimensional coordinate system 306.

In FIG. 5, the coordinates of the points Q₁ to Q₉ in the two-dimensionalcoordinate system set for the display screen displayed by the imagedisplay element 108 are Q₁(x1, y1)^(T), Q₂(x2, y2)^(T), . . . , Q₉(x9,y9)^(T), respectively. The coordinates are, for example, a number of apixel located at a center of each point. Further, the center points P ofthe cornea 302 of the user 300 when the user 300 gazes at the points Q1to Q9 are labeled P₁ to P₉. In this case, the coordinates of the pointsP1 to P9 in the two-dimensional coordinate system 306 are P₁(X1,Y1)^(T), P₂(X2, Y2)^(T), . . . , P₉(X9, Y9)^(T). T represents atransposition of a vector or a matrix.

A matrix M with a size of 2×2 is defined as Equation (1) below.

$\begin{matrix}{M = \begin{pmatrix}m_{11} & m_{12} \\m_{21} & m_{22}\end{pmatrix}} & (1)\end{matrix}$

In this case, when the matrix M satisfies Equation (2) below, the matrixM is a matrix for projecting the gaze direction of the user 300 onto animage plane that is displayed by the image display element 108.

Q _(N) =MP _(N)(N=1, . . . ,9)  (2)

When Equation (2) is written specifically, Equation (3) below isobtained.

$\begin{matrix}{\begin{pmatrix}x_{1} & x_{2} & \ldots & x_{9} \\y_{1} & y_{2} & \ldots & y_{9}\end{pmatrix} = {\begin{pmatrix}m_{11} & m_{12} \\m_{21} & m_{22}\end{pmatrix}\begin{pmatrix}X_{1} & X_{2} & \ldots & X_{9} \\Y_{1} & Y_{2} & \ldots & Y_{9}\end{pmatrix}}} & (3)\end{matrix}$

By transforming Equation (3), Equation (4) below is obtained.

$\begin{matrix}{\begin{pmatrix}x_{1} \\x_{2} \\\vdots \\x_{9} \\y_{1} \\y_{2} \\\vdots \\y_{9}\end{pmatrix} = {\begin{pmatrix}X_{1} & Y_{1} & 0 & 0 \\X_{2} & Y_{2} & 0 & 0 \\\vdots & \vdots & \vdots & \vdots \\X_{9} & Y_{9} & 0 & 0 \\0 & 0 & X_{1} & Y_{1} \\0 & 0 & X_{2} & Y_{2} \\\vdots & \vdots & \vdots & \vdots \\0 & 0 & X_{9} & Y_{9}\end{pmatrix}\begin{pmatrix}m_{11} \\m_{12} \\m_{21} \\m_{22}\end{pmatrix}}} & (4)\end{matrix}$

Here, If

$\begin{matrix}{{y = \begin{pmatrix}x_{1} \\x_{2} \\\vdots \\x_{9} \\y_{1} \\y_{2} \\\vdots \\y_{9}\end{pmatrix}},{A = {{\begin{pmatrix}X_{1} & Y_{1} & 0 & 0 \\X_{2} & Y_{2} & 0 & 0 \\\vdots & \vdots & \vdots & \vdots \\X_{9} & Y_{9} & 0 & 0 \\0 & 0 & X_{1} & Y_{1} \\0 & 0 & X_{2} & Y_{2} \\\vdots & \vdots & \vdots & \vdots \\0 & 0 & X_{9} & Y_{9}\end{pmatrix}x} = \begin{pmatrix}m_{11} \\m_{12} \\m_{21} \\m_{22}\end{pmatrix}}},} & \;\end{matrix}$

Equation (5) below is obtained:

y=Ax  (5)

In Equation (5), elements of the vector y are known since these arecoordinates of the points Q₁ to Q₉ that are displayed on the imagedisplay element 108 by the gaze detection unit 202. Further, theelements of the matrix A can be acquired since the elements arecoordinates of a vertex P of the cornea 302 of the user 300. Thus, thegaze detection unit 202 can acquire the vector y and the matrix A. Avector x that is a vector in which elements of a transformation matrix Mare arranged is unknown. Since the vector y and matrix A are known, anissue of estimating matrix M becomes an issue of obtaining the unknownvector x.

Equation (5) becomes the main issue to decide when the number ofequations (that is, the number of points Q presented to the user 300 bythe gaze detection unit 202 at the time of calibration) is larger thanthe number of unknown numbers (that is, the number 4 of elements of thevector x). Since the number of equations is nine in the exampleillustrated in Equation (5), Equation (5) is the main issue to decide.

An error vector between the vector y and the vector Ax is defined asvector e. That is, e=y−Ax. In this case, a vector x_(opt) that isoptimal in the sense of minimizing the sum of squares of the elements ofthe vector e can be obtained from Equation (6) below.

x _(opt)=(A ^(T) A)⁻¹ AT _(y)  (6)

Here, “−1” indicates an inverse matrix.

The gaze detection unit 202 uses the elements of the obtained vectorx_(opt) to constitute the matrix M of Equation (1). Accordingly, usingthe coordinates of the vertex P of the cornea 302 of the user 300 andthe matrix M, the gaze detection unit 202 estimates a point at which theright eye of the user 300 is gazing in the moving image displayed by theimage display element 108 according to Equation (2). Here, the gazedetection unit 202 further receives distance information between the eyeof the user and the image display element 108 from the head mounteddisplay 100 and corrects an estimated coordinate value at which the usergazes according to the distance information. It should be noted that adeviation in the estimation of a gaze position according to the distancebetween the eye of the user and the image display element 108 may beignored as an error range. Accordingly, the gaze detection unit 202 cancalculate a right-eye gaze vector connecting the gaze point of the righteye on the image display element 108 and the vertex of the cornea of theright eye of the user. Similarly, the gaze detection unit 202 cancalculate a left-eye gaze vector connecting the gaze point of the lefteye on the image display element 108 and the vertex of the cornea of theleft eye of the user. It should be noted that it is possible to specifythe gazing point of the user on the two-dimensional plane with the gazevector of only one eye, and calculate depth direction information of thegaze point of the user by obtaining the gaze vectors of both eyes. Thus,the gaze detection device 200 can specify the gaze point of the user. Itshould be noted that the method of specifying the gaze point shownherein is merely an example, and a gaze point of the user may bespecified using a different method.

The determination unit 203 determines whether or not the motion of theuser acquired from the head mounted display 100 via the secondcommunication unit 22 satisfies a predetermined “switching condition” inwhich the user does not recognize switching between the images even whenthe moving image displayed on the display unit 121 is switched fromfirst moving image data to second moving image data which is differentfrom the first moving image data and of which a story is developed fromthe first moving image. That “the story is developed” refers to the factthat a flow of story of the moving image is changed to a different flowherein.

Here, when a predetermined “determination condition” is satisfied, thedetermination unit 203 may determine whether or not the “switchingcondition” is satisfied. When the user views a specific place in themoving image displayed on the display unit 121, the determination unit203 determines that the “determination condition” for start of thedetermination process is satisfied and determines whether or not the“switching condition” is satisfied. For example, it is said that the“determination condition” is satisfied when the user views a specificplace continuously for a predetermined time or more or when the userviews a specific place for a total predetermined time or more within acertain time. The “specific place” is a range that is identified bycoordinates or a plurality of sets of coordinates.

FIG. 7A is an example of the determination condition data 212 regardingthe determination condition. For example, in the image display system 1,when a condition “the user views the region A” is satisfied whilecertain first moving image data is displayed on the display unit 121,switching to an image A as the second moving image data then occurs.Further, in the image display system 1, when a condition “the user viewsthe region B” is satisfied while the first moving image data isdisplayed on the display unit 121, switching to a moving image B as thesecond moving image data then occurs. It should be noted that, in FIG.7A, there are two patterns, that is, a case of switching to the movingimage A as the second moving image and a case of switching to the movingimage B as the second moving image, but the present invention is notlimited thereto. For example, the second moving image to be switched maybe one pattern of only the moving image A or may have a number ofpatterns.

Further, the image display system 1 switches from the first moving imagedata to the second moving image data at a timing when it is determinedthat the “switching condition” in which it is determined that the userdoes not recognize switching between the images even when switching fromthe first moving image data to the second moving image data occurs issatisfied, instead of switching to the second moving image dataimmediately after the user views the region A or the region B.

Further, although not illustrated in FIG. 7A, when a plurality ofconditions are satisfied, it may be determined that the “switchingcondition” is satisfied. For example, when the user views the region Aand views the region B, the image display system 1 may then switch tothe moving image C. In this case, the image display system 1 switches tothe moving image C when the user views the region B after viewing theregion A, but the image display system 1 may switch to the moving imageD when the user views the region A after viewing the region B.

For example, the determination unit 203 determines whether or not themotion of the gaze detected by the gaze detection unit 202 satisfies the“switching condition”. That is, when the user gazes at one point, theuser easily notices that the image has been switched. On the other hand,when a position of the gaze is not fixed, the user does not easilynotice that the image has been switched. Therefore, the gaze detectiondevice 200 determines whether or not the image is to be switchedaccording to a degree of motion of the gaze of the user.

Therefore, for example, the “switching condition” is a condition fordetermining that the moving image to be displayed on the display unit121 is switched from the first moving image data to the second movingimage data when a total moving distance according to the motion of thegaze of the user detected by the gaze detection unit 202 is equal to orgreater than a predetermined value within a predetermined time or whenit is detected that the user is viewing a plurality of places separateby a predetermined distance or more within a predetermined time.

Specifically, (1) when the gaze detection device 200 detects coordinatesat which the user gazes in the moving image at a certain time t0, thegaze detection device 200 stores the coordinates C0

(2) When the gaze detection device 200 detects new coordinates at whichthe user gazes in the moving image at t1 after a predetermined time fromt0, the gaze detection device 200 stores the coordinates C1.

(3) An amount of movement M of the coordinates C1 from the coordinatesC0 is obtained, and a movement speed S [°/sec] of the motion of the gazeof the user is obtained using a difference time (t1−t0) between t0 andt1, as shown in the following equation.

$\begin{matrix}{S = {2\; {{{atan}\left( \frac{M}{2L} \right)}/\left( {{t\; 1} - {t\; 0}} \right)}}} & \left\lbrack {{Math}.\mspace{14mu} 5} \right\rbrack\end{matrix}$

Here, L is an optical distance between the display and the eyeball.

(4) When the speed S exceeds a predetermined value, the time t1−t0 isadded as a movement time Ti to an addition value Ta, and when the speedS does not exceed the predetermined value, the movement time Ti iscleared to 0.

(5) (1) to (4) are repeated during a predetermined “switching period”determined as a scene change period in advance. When the addition valueTa of the movement time Ti exceeds the predetermined threshold value,the gaze of the user is regarded as being not determined, and scenechange is performed. That is, the moving image is switched.

For example, when the gaze is moving at a speed of 60°/sec, a movingfield of view is regarded as being sufficiently narrow, and switching ofthe moving image is considered at a timing when this speed continues for0.5 seconds. When the switching period is 3.3 seconds and coordinates ofthe gaze of the user can be detected at 60 Hz, sampling can be performedapproximately 200 times (3.3 sec/( 1/60 Hz)=198 times). When the gazespeed is detected as ≥60°/sec 30 consecutive times (0.5 sec/( 1/60Hz)=30 times) at the time of this sampling, a total movement time Tabecomes 0.5, and this condition is regarded as being satisfied. It canbe determined that the switching condition is satisfied.

Further, for example, when the determination unit 203 acquires themotion of the head of the user detected by the motion sensor 125 fromthe head mounted display 100 via the second communication unit 22, thedetermination unit 203 determines whether or not the motion of the headof the user satisfies the “switching condition.” That is, when the useris stationary, the user easily notices that the image has been switched.On the other hand, when the motion of the user is intense to someextent, it is difficult for the user to notice that the image has beenswitched. Therefore, the gaze detection device 200 determines whether ornot the image is switched according to the motion of the head of theuser that can be detected by the head mounted display 100.

Therefore, for example, the “switching condition” is a condition that,when the amount of movement of the user, that is, the motion of theuser's head within a predetermined time, exceeds a predetermined value,it is determined that the moving image to be displayed on the displayunit 121 is switched from the first moving image data to the secondmoving image data.

Specifically, (1) when a rotation angle (roll, pitch, and yaw) that is aposture of the head mounted display 100 mounted on the head of the userdetected by the motion sensor 125 is input from the head mounted display100 at a certain time t0, the gaze detection device 200 stores a postureR0 thereof. R0 is a three-dimensional vector quantity.

(2) When a new posture of the head mounted display 100 is input from thehead mounted display 100 at t1 which is a predetermined time after t0,the gaze detection device 200 stores a posture R1 thereof. R1 is athree-dimensional vector quantity.

(3) A three-dimensional vector M which is the amount of movement of theposture R1 from the posture R0 is obtained and divided by a differencetime (t1−t0) between t0 and t1 to thereby obtain an angular velocity(M/(t1−t0)) of rotation of the head mounted display 100 mounted on theuser. Alternatively, when a gyro sensor is included in the head mounteddisplay 100, the angular velocity obtainable from the gyro sensor can beused as a value of S as it is.

(4) The magnitude of the angular velocity vector S is a one-dimensionalamount with the L2 norm ∥S∥ or the like, and when a value thereofexceeds a predetermined value, the time of t1−t0 is added as themovement time Ti to the addition value Ta, and when the value does notexceed the predetermined value, the movement time Ti is cleared to 0.

(5) (1) to (4) are repeated during the predetermined “switching period,”and when the addition value Ta of the movement time Ti exceeds thepredetermined threshold value, the gaze of the user is regarded as notbeing determined and the scene change is performed. That is, the movingimage is switched.

For example, when the head of the user is moving at a speed of 60°/sec,the moving field of view is regarded as being sufficiently narrow, andswitching of the moving image is considered at a timing when this speedcontinues for 0.5 seconds. When it is assumed that the switching periodis 3.3 seconds and the detection of the angular velocity of the headmounted display 100 can be performed at 60 Hz, sampling can be performedapproximately 200 times (3.3 sec/( 1/60 Hz)=198 times). At the time ofthis sampling, when the magnitude of the angular velocity ∥S∥≥60°/sec isdetected continuously 30 times (0.5 sec/( 1/60 Hz)=30 times), the totalmovement time Ta becomes 0.5, and this condition is regarded as beingsatisfied. It can be determined that the switching condition issatisfied. Here, ∥S∥ can be calculated as a value obtained by squaringthe angular velocity of roll, pitch, and yaw and taking a square root.

Further, for example, when the determination unit 203 acquires theposition of the head of the user detected by the position sensor 126from the head mounted display 100 via the second communication unit 22,the determination unit 203 determines whether or not the position of thehead of the user satisfies the “switching condition.” That is, asdescribed, when the user is stationary, the user easily notices that theimage has been switched. On the other hand, when the motion of the useris intense to some extent, it is difficult for the user to notice thatthe image has been switched. Therefore, the gaze detection device 200determines whether or not the image is switched according to the motionof the head of the user that can be detected by the head mounted display100.

Therefore, for example, the “switching condition” is a condition that,when the amount of movement of the user, that is, the motion of theuser's head within a predetermined time exceeds a predetermined value,it is determined that the moving image to be displayed on the displayunit 121 is switched from the first moving image data to the secondmoving image data.

Specifically, (1) when a three-dimensional coordinate position (x, y, z)that is a position of the head mounted display 100 mounted on the headof the user detected by the position sensor 126 is input from the headmounted display 100 at a certain time t0, the gaze detection device 200stores the coordinate position R0. R0 is a three-dimensional vectorquantity.

(2) When a new position of the head mounted display 100 is input fromthe head mounted display 100 at t1 which is a predetermined time aftert0, the gaze detection device 200 stores a coordinate position R1thereof. R1 is a three-dimensional vector quantity.

(3) The amount of movement M which is the coordinate position R1 fromthe coordinate position R0 is obtained and divided by a difference time(t1−t0) between t0 and t1 to thereby obtain a three-dimensional vectorS(M/(t1−t0)) of a movement speed of the position of the head mounteddisplay 100 mounted on the user.

(4) The magnitude of the movement speed S is a one-dimensional amountwith the L2 norm ∥S∥ or the like, and when a value thereof exceeds apredetermined value, the time of t1−t0 is added as the movement time Tito the addition value Ta, and when the value does not exceed thepredetermined value, the movement time Ti is cleared to 0.

(5) (1) to (4) are repeated during the predetermined “switching period,”and when the addition value Ta of the movement time Ti exceeds thepredetermined threshold value, the gaze of the user is regarded as notbeing determined and the scene change is performed. That is, the movingimage is switched.

For example, when the head of the user is moving at a speed of 10cm/sec, the moving field of view is regarded as being sufficientlynarrow, and switching of the moving image is considered at a timing whenthis speed continues for 0.5 seconds. When it is assumed that theswitching period is 3.3 seconds and the detection of the angularvelocity of the head mounted display 100 can be performed at 60 Hz,sampling can be performed approximately 200 times (3.3 sec/( 1/60Hz)=198 times). At the time of this sampling, when the speed magnitude∥S∥≥10 cm/sec is detected continuously 30 times (0.5 sec/( 1/60 Hz)=30times), the total movement time Ta becomes 0.5, and this condition isregarded as being satisfied. It can be determined that the switchingcondition is satisfied. Here, ∥S∥ can be calculated as a value obtainedby squaring a speed of (x, y, z) in a three-dimensional coordinate spaceand taking a square root.

In this case, when the scene of the moving image displayed on thedisplay unit 121 is a predetermined “switching period,” thedetermination unit 203 determines whether or not the “switchingcondition” is satisfied. That is, the timing at which switching from thefirst moving image data to the second moving image data is possible isdetermined in advance according to the story. Therefore, thedetermination unit 203 does not need to constantly determine the“switching condition” and may determine the “switching condition” onlyin the “switching period,” which is a timing at which switching from thefirst moving image data to the second moving image data is possible.

When the determination unit 203 determines that the “switchingcondition” is satisfied, the switching unit 204 switches the movingimage displayed on the display unit 121 from the first moving image datato the second moving image data. In this case, the switching unit 204switches the moving image from the image data 211 stored in the storagedevice 21 to a moving image with a pattern associated with thedetermination result of the determination unit 203.

The difference calculation unit 205 obtains the difference between thefirst moving image data and the second moving image data. Here, thedifference calculation unit 205 obtains a difference between the firstmoving image data and the second moving image data while the switchingfrom the first moving image data to the second moving image data ispossible.

Here, when the determination unit 203 determines that the “determinationcondition” is satisfied, the display processing unit 201 startsreproduction of the second moving image data in the background.Therefore, the difference calculation unit 205 compares the first movingimage data being displayed on the head mounted display 100 with thesecond moving image data being reproduced in the background to obtainthe difference.

The determination unit 203 determines whether or not the differencecalculated by the difference calculation unit 205 matches the “switchingcondition” in which even when the moving image displayed on the displayunit 121 is switched from the first moving image data to the secondmoving image data different from the first moving image, the user doesnot recognize the switching between the images. The “switchingcondition” in this case is a condition for determining that the image isswitched in which it is made difficult for the user to notice switchingeven when the first moving image is switched to the second moving imagein a case in which the difference is within a predetermined range. Forexample, when a change in the moving image is small to some extent, theuser is regarded as not gazing at a certain place, but moving his or hergaze, and the moving image is switched. In addition, when the change inthe moving image is large to some extent, the user is regarded as notgazing at a certain place, but moving his or her gaze greatly accordingto the change, and the moving image is switched.

Specifically, (1) the gaze detection device 200 calculates (1) a featureamount of the first moving image and a feature amount of the secondmoving image. An example of a method of obtaining the feature amountincludes a method of (a) adding luminance values of an image at acertain time t0 and an image at t1 after a predetermined time for thefirst moving image and the second moving image, (b) obtaining featurepoints using an algorithm such as Oriented Fast and Rotated BRIEF (ORB)or accelerated KAZE (AKAZE) for the first moving image and the secondmoving image and obtaining a sum of root-mean-squares of amounts ofmovement of the feature points from the image at the certain time t0 tot1 that is after a predetermined time, and (c) obtaining the amount ofmovement from an optical flow from the certain time t0 to the time t1which is after a predetermined time for the first moving image and thesecond moving image.

(2) When the obtained value V does not exceed a predetermined value, atime t1−t0 is added as a movement time Ti to the addition value Ta, andwhen the obtained value V exceeds the predetermined value, Ti iscleared.

(3) (1) to (2) are repeated for a predetermined “switching period,” andwhen the addition value Ta of the movement time Ti exceeds apredetermined upper limit threshold value or when the addition value Tafalls below a lower limit threshold value, a scene change is performed.That is, the moving image is switched. A case in which the additionvalue Ta exceeds the upper limit threshold value means that a change inan object in the moving image is extremely large, and when the object isintensively changing so that the change cannot be traced with the eyesof the user, it is difficult for the user to recognize switching evenwhen the switching is performed. In addition, a case in which the valuefalls below the lower limit threshold value means that the change in theobject in the moving image is small and the object is substantiallystationary.

For example, switching the moving image at a timing when the videochanges greatly continuously for 0.5 second is considered. When it isassumed that the switching period is 3.3 seconds and the calculation ofthe feature amount is performed at 60 Hz, approximately 200 calculationsare possible during the switching period (3.3 sec/( 1/60 Hz)=198 times).When it is detected that the amount of change in the feature amountexceeds the threshold value continuously 30 times (0.5 sec/( 1/60 Hz)=30times) during this period, the total movement time Ta becomes 0.5, andthis condition is regarded as being satisfied. It can be determined thatthe switching condition is satisfied.

For example, the switching condition described above is stored in thestorage device 21 as the switching condition data 213 as shown by way ofexample in FIG. 7B. Specifically, the switching condition data 213includes a switchable “timing” and a “threshold value” that is used as aswitching condition. In the example illustrated in FIG. 7B, theswitchable timing is a “period α.” Further, the switching conditionincludes a “first threshold value” that is used for a determination ofthe motion of the gaze detected by the gaze detection unit 202, a“second threshold value” that is used for a determination of the motionof the head of the user detected by the motion sensor 125 or the motionof the head obtained from the position detected by the position sensor126, and a “third threshold value” that is used for a determination ofan image change amount obtained by the difference calculation unit 205.

As described above, the gaze detection device 200 can determine “1.Switch when the motion of the gaze of the user or the motion of the headis intense,” “2. Switch when a difference between moving images isextremely large or small,” or the like as the “switching condition.”Further, the gaze detection device 200 can determine “3. Switchaccording to a combination of the above conditions 1 and 2” as“switching condition.”

<Operation>

Hereinafter, an operation of the image display system 1 will bedescribed with reference to the flowchart illustrated in FIG. 8. In theimage display system 1, first, a first moving image is reproduced anddisplayed (S01).

During reproduction of the first moving image data, the determinationunit 203 determines whether or not the determination condition issatisfied (S02). For example, in the example of the determinationcondition data 212 illustrated in FIG. 7A, a determination is made thatthe condition is satisfied when “region A is viewed” or “region B isviewed” in a certain time range of the moving image. FIG. 9A illustratesan example of the moving image displayed on the display unit 121. In theexample of FIG. 9A, a region on the upper side of the door is set asregion A, and a region on the lower side of the door is set as region B.In the example of the determination condition data 212 of FIG. 7A andthe example of the moving image data of FIG. 9A, when the user views theregion on the upper side of the door (region A), the moving image A isselected as the second moving image data. On the other hand, when theuser views the region on the lower side of the door (region B), themoving image B is selected as the second moving image data.

When the image display system 1 determines that the determinationcondition is satisfied (YES in S02), the image display system 1 startsthe reproduction of the second moving image in the background (S03).Specifically, the image display system 1 starts the reproduction of thesecond moving image in the background with the start of the switchingperiod, as shown in the time chart of FIG. 10A. In the example describedabove with reference to FIGS. 7A and 9A, when the user views the regionA, the moving image A is set as the second moving image, and backgroundreproduction is started. On the other hand, when the user views theregion B, the moving image B is set as the second moving image, andbackground reproduction is started. The switching period is stored, forexample, in the switching condition data 213. In the example of theswitching condition data 213 illustrated in FIG. 7B, the period α is setas the switching period.

In addition, when the switching period starts (YES in S04), the imagedisplay system 1 determines whether or not the switching condition issatisfied, that is, determines whether a motion of the gaze of the useris equal to or greater than the threshold value or whether a motion ofthe head of the user is equal to or greater than a threshold value(S05).

When the motion of the gaze of the user is not equal to or greater thanthe threshold value and the motion of the head of the user is not equalto or greater than the threshold value, the image display system 1determines whether or not the switching condition is satisfied, that is,whether or not a difference between the first moving image and thesecond moving image is outside a range of a threshold value (S06).

When the motion of the gaze of the user is equal to or greater than thethreshold value, when the motion of the head of the user is equal to orgreater than the threshold value, or when the difference between thefirst moving image and the second moving image is outside the range ofthe threshold value (YES in S05 or YES in S06), a display on the displayunit 121 is switched from the first moving image to the second movingimage, and the reproduction of the first moving image is stopped (S07).For example, when the moving image A is selected as the second image inthe example illustrated in FIGS. 7A and 9A, the moving image A asillustrated in FIG. 9B is displayed as the second image. Further, whenthe moving image B is selected as the second image, the moving image Bas illustrated in FIG. 9C is displayed as the second image.

When the first moving image and the second moving image are within therange of the threshold value (NO in S07), the processes of steps S05 toS06 are repeated until the switching period ends.

Further, the switching condition data 213 of FIG. 7C and the time chartof FIG. 10B are examples in which the first switching period (period α)and the second switching period (period β) are set twice. Thus, when aplurality of switching periods are set, the processes of steps S04 toS06 are repeated. In the case in which the switching period is set aplurality of times, the image is switched during the second switchingperiod when the switching condition is satisfied in the second switchingperiod even when the switching condition is not satisfied in the firstswitching period, as illustrated in FIG. 10B.

It should be noted that, in the above-described example, when the motionof the gaze is equal to or greater than the threshold value or when themotion of the head is equal to or greater than the threshold value, thedisplay is switched to display of the second moving image and thereproduction of the first moving image is stopped. Here, it ispreferable for the present invention to be able to be realized, forexample, for a head mounted display 100 that does not include theimaging unit 124 for detecting the gaze or a head mounted display 100that does not include the motion sensor 125 or the position sensor 126for detecting the motion. Therefore, the determination in thedetermination unit 203 is determined according to the configuration ofthe head mounted display 100. That is, when the image display system 1cannot detect the gaze, the motion of the gaze is not used for thedetermination. Further, when the image display system 1 cannot performmotion detection or position detection, the motion of the head is notused for the determination.

Thus, in the image display system 1 according to the embodiment, thestory selected from the stories of a plurality of patterns is developedaccording to a situation of the user. Accordingly, the user can enjoythe story suitable for the user. Further, when the story is switched,the switching is performed so that the user does not notice theswitching between images according to a state of the gaze of the user, astate of the motion of the head, a state of the image, and the like.Accordingly, the user can concentrate on and enjoy the moving imagesdisplayed on the image display system 1 without feeling stress caused byswitching between the images.

<Supplements>

It is apparent that the image display system 1 is not limited to theabove-described configuration or the like but may be realized by anotherscheme for realizing the spirit of the invention. Hereinafter, otherexamples which can be included as the spirit of the present inventionwill be described.

(1) There is one image display element 108, but the present invention isnot limited thereto. The head mounted display 100 may include two imagedisplay elements including an image display element for the left eye ofthe user 300 and an image display element for the right eye of the user300, which may be driven separately. Accordingly, fine control such asfocus adjustment according to visual acuity of the left and right eyesof the user 300 can be performed.

(2) In order to detect the gaze of the user 300, the video reflected bythe hot mirror 112 is imaged as a scheme of imaging the eyes of the user300. Here, the eyes of the user 300 may be directly imaged withoutpassing through the hot mirror 112.

(3) The scheme related to gaze detection is an example, and the gazedetection method using the head mounted display 100 and the gazedetection device 200 is not limited thereto.

First, the example in which a plurality of infrared light sources thatradiate near-infrared light as invisible light are provided is shown,but a scheme of irradiating the eye of the user with near-infrared lightis not limited thereto. For example, a configuration in which pixelsincluding sub pixels that emit near infrared light are provided forpixels constituting the image display element 108 of the head mounteddisplay 100 may be adopted, the sub pixels that emit near infrared lightmay be caused to selectively emit the near infrared light, and the eyesof the user may be irradiated with the near-infrared light. Further,alternatively, a configuration in which a retinal projection display isincluded in the head mounted display 100 instead of the image displayelement 108, and pixels that emit light with near infrared light colorare included in an image displayed on the retinal projection display andprojected on a retina of the user to realize irradiation with nearinfrared light may be adopted. Even in the case of the image displayelement 108 or the case of the retinal projection display, sub-pixelsthat emit near-infrared light may be periodically changed.

In addition, the above-described eye-gaze detection algorithm is notlimited to the above-described scheme, and other algorithms may be usedas long as the gaze detection can be realized.

(4) Further, although changing of the advertisement displayed on thehead mounted display 100 is realized by the processors of the headmounted display 100 and the gaze detection device 200 executing theimage display program P and the like, this may be realized by a logicalcircuit (hardware) form of an integrated circuit (an integrated circuit(IC) chip, large scale integration (LSI), or the like) or a dedicatedcircuit in the gaze detection device 200. Further, the circuit may berealized by one or a plurality of integrated circuits, or the functionsof the plurality of functional units described above may be realized byone integrated circuit. The LSI may be called VLSI, super LSI, ultraLSI, or the like according to an integration difference. That is, asillustrated in FIG. 11, the head mounted display 100 may include a firstcommunication circuit 110 a, a communication control circuit 118 a, adisplay circuit 121 a, an infrared light irradiation circuit 122 a, animage processing circuit 123 a, an imaging circuit 124 a, a motiondetection circuit 125 a, and a position detection circuit 126 a, andrespective functions thereof are the same as those of respective unitshaving the same name described above. Further, the gaze detection device200 may include a second communication circuit 22 a, an input circuit 23a, an output circuit 24 a, a display processing circuit 201 a, a gazedetection circuit 202 a, a determination circuit 203 a, a switchingcircuit 204 a, and a difference calculation circuit 205 a, andrespective functions thereof are the same as those of respective unitshaving the same name described above.

Further, the image display program P may be recorded on aprocessor-readable recording medium, and the recording medium may be a“non-transitory tangible medium” such as a tape, a disk, a card, asemiconductor memory, or a programmable logic circuit. Further, theimage display program P may be supplied to the processor through anytransmission medium (such as a communication network or broadcast waves)capable of transmitting the image display program P. The presentinvention can also be realized in the form of a data signal embodied ina carrier wave, in which the image display program P is implemented byelectronic transmission.

It should be noted that the image display program P may be installedusing, for example, a script language such as ActionScript, JavaScript(registered trademark), Python, or Ruby, a compiler language such as a Clanguage, C++, C#, Objective-C, or Java (registered trademark), or thelike.

(5) The respective configurations and respective supplements may beappropriately combined.

The present invention has an effect of being able to provide a movingimage with story development according to users, and can be used forvarious image display devices, head mounted displays, and the like.

EXPLANATION OF REFERENCES

-   -   1 Image display system    -   100 Head mounted display    -   118 First communication unit    -   121 Display unit    -   122 Infrared light irradiation unit    -   123 Image processing unit    -   124 Imaging unit    -   125 Motion sensor    -   126 Position sensor    -   200 Gaze detection device    -   201 Display processing unit    -   202 Gaze detection unit    -   203 Determination unit    -   204 Switching unit    -   205 Difference calculation unit    -   21 Storage device    -   211 Image data    -   212 Determination condition data    -   213 Switching condition data    -   22 Second communication unit (acquisition unit)    -   23 Input device    -   24 Output device

1. An image display system comprising: a display unit that displays amoving image; an acquisition unit that acquires a motion of a userviewing the moving image displayed by the display unit; a determinationunit that determines whether or not the motion of the user acquired bythe acquisition unit satisfies a predetermined switching condition inwhich the user does not recognize switching between images even when themoving image displayed by the display unit is switched from first movingimage data to second moving image data different from the first movingimage; and a switching unit that switches the moving image displayed bythe display unit from the first moving image data to the second movingimage data when the determination unit determines that the switchingcondition is satisfied.
 2. The image display system according to claim1, further comprising: an imaging unit that captures an image of an eyeof the user viewing the moving image displayed by the display unit; anda gaze detection unit that detects a gaze of the user as a motion of theuser when the acquisition unit acquires the captured image of the eye ofthe user acquired by the imaging unit, wherein the determination unitdetermines whether or not the motion of the gaze detected by the gazedetection unit satisfies the switching condition.
 3. The image displaysystem according to claim 2, wherein, when the user views a specificplace in the first moving image, the determination unit determineswhether or not the switching condition is satisfied.
 4. The imagedisplay system according to claim 1, further comprising a motion sensorthat detects a motion of a head of the user, wherein, when theacquisition unit acquires the motion of the head of the user detected bythe motion sensor, the determination unit determines whether or not themotion of the head of the user satisfies the switching condition.
 5. Theimage display system according to claim 1, further comprising a positionsensor that detects a position of a head of the user, wherein, when theacquisition unit acquires the position of the head of the user detectedby the position sensor, the determination unit determines whether or notthe position of the head of the user satisfies the switching condition.6. The image display system according to claim 1, further comprising: adifference calculation unit that obtains a difference between the firstmoving image data and the second moving image data, wherein thedetermination unit determines whether or not the difference calculatedby the difference calculation unit matches a defined switching conditionin which even when the moving image displayed on the display unit isswitched from the first moving image data to the second moving imagedata different from the first moving image, the user does not recognizethe switching between the images.
 7. The image display system accordingto claim 1, further comprising a moving image storage unit that storesmoving image data of a story with a plurality of patterns associatedwith a determination condition, wherein the determination unitdetermines whether or not the determination condition is satisfied whena scene of the moving image displayed by the display unit is within arange of a defined period, and the switching unit performs switchingfrom the moving image data stored in the moving image storage unit tothe moving image with the pattern associated with the determinationcondition determined by the determination unit.
 8. The image displaysystem according to claim 2, wherein the switching condition is acondition for determining that switching is performed when a totalmoving distance according to the motion of the gaze of the user is equalto or greater than a predetermined value within a predetermined time orwhen it is detected that the user is viewing a plurality of placesseparated by a predetermined distance or more within a predeterminedtime.
 9. The image display system according to claim 4, wherein theswitching condition is a condition for determining that switching isperformed when the amount of movement of the user within a predeterminedtime exceeds a predetermined value.
 10. An image display methodcomprising: a display step of displaying a moving image on a displayunit; an acquisition step of acquiring a motion of a user viewing themoving image displayed in the display step; a determination step ofdetermining whether or not the motion of the user acquired in theacquisition step satisfies a predetermined switching condition in whichthe user does not recognize switching between the images even when themoving image displayed by the display unit is switched from first movingimage data to second moving image data different from the first movingimage; and a switching step of switching the moving image displayed bythe display unit from the first moving image data to the second movingimage data when it is determined in the determination step that theswitching condition is satisfied.
 11. An image display program causing acomputer to function as: a display function of displaying a moving imageon a display unit; an acquisition function of acquiring a motion of auser viewing the moving image displayed through the display function; adetermination function of determining whether or not the motion of theuser acquired through the acquisition function satisfies a predeterminedswitching condition in which the user does not recognize switchingbetween the images even when the moving image displayed by the displayunit is switched from first moving image data to second moving imagedata different from the first moving image; and a switching function ofswitching the moving image displayed by the display unit from the firstmoving image data to the second moving image data when it is determinedin the determination function that the switching condition is satisfied.